This specification relates to performing neural network computations in hardware.
Neural networks are machine learning models that employ one or more layers of models to generate an output, e.g., a classification, for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to one or more other layers in the network, i.e., one or more other hidden layers or the output layer of the network. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.
Some neural networks include a depth concatenation layer that receives as input two or more tensors, i.e., multi-dimensional matrices, that are the outputs of other layers in the neural network and concatenates the input tensors along the depth dimension. In particular, each input tensor has two spatial dimensions x and y and a depth dimension z. By concatenating two input tensors, one having dimensions x1 by y1 by z1 and the other having dimensions x1 by y1 by z2, along the depth dimension z, the depth concatenation layer generates an output tensor that has dimensions x1 by y1 by (z1+z2). The output of the depth concatenation layer can then be provided as input to one or more other layers in the neural network or used as the output of the neural network.